Matrix Algebra Fundamentals

Dr. Lucy D’Agostino McGowan

What is a Matrix?

  • A matrix is a rectangular array of numbers arranged in rows and columns
  • Written using square brackets or parentheses
  • Each number in the matrix is called an element or entry

What is a Matrix?

\[\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}\]

Matrix Dimensions

Rows are horizontal ↔︎

Columns are vertical ↕

\[ \mathbf{A} = \begin{bmatrix} \require{color}\colorbox{#eec8e1}{$a_{11}$} & \colorbox{#eec8e1}{$a_{12}$} & \colorbox{#eec8e1}{$a_{13}$} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Row 1} \]

Matrix Dimensions

Rows are horizontal ↔︎

Columns are vertical ↕

\[ \mathbf{A} = \begin{bmatrix} \require{color}a_{11} & a_{12} & a_{13} \\ \colorbox{#eec8e1}{$a_{21}$} & \colorbox{#eec8e1}{$a_{22}$} & \colorbox{#eec8e1}{$a_{23}$} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Row 2} \]

Matrix Dimensions

Rows are horizontal ↔︎

Columns are vertical ↕

\[ \mathbf{A} = \begin{bmatrix} \require{color}a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ \colorbox{#eec8e1}{$a_{31}$} & \colorbox{#eec8e1}{$a_{32}$} & \colorbox{#eec8e1}{$a_{33}$} \end{bmatrix} \leftarrow \text{Row 3} \]

Matrix Dimensions

Rows are horizontal ↔︎

Columns are vertical ↕

\[ \mathbf{A} = \begin{bmatrix} \colorbox{#eec8e1}{$a_{11}$} & a_{12} & a_{13} \\ \colorbox{#eec8e1}{$a_{21}$} & a_{22} & a_{23} \\ \colorbox{#eec8e1}{$a_{31}$} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Column 1} \]

Matrix Size: n × p

A matrix with n rows and p columns is called an n × p matrix

\[\mathbf{A}_{3 \times 3} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}\]

This is a 3 × 3 matrix (3 rows, 3 columns)

Matrix Size: n × p

A matrix with n rows and p columns is called an n × p matrix

\[\mathbf{B}_{2 \times 4} = \begin{bmatrix} b_{11} & b_{12} & b_{13} & b_{14} \\ b_{21} & b_{22} & b_{23} & b_{24} \end{bmatrix}\]

This is a 2 × 4 matrix (2 rows, 4 columns)

Examples of Different Matrix Sizes

Row vector (1 × p): \[\mathbf{r} = \begin{bmatrix} 1 & 3 & 5 & 7 \end{bmatrix}\]

Column vector (n × 1): \[\mathbf{c} = \begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix}\]

Examples of Different Matrix Sizes

Square matrix (n × n): \[\mathbf{S} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\]

Matrix Addition

Matrices can be added only if they have the same dimensions

Add corresponding elements: \[\mathbf{A} + \mathbf{B} = \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\ a_{21} + b_{21} & a_{22} + b_{22} \end{bmatrix}\]

Matrix Addition

Example: \[\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}\]

\[= \begin{bmatrix} \color{red}{1+5} & \color{blue}{2+6} \\ \color{green}{3+7} & \color{purple}{4+8} \end{bmatrix} = \begin{bmatrix} \color{red}{6} & \color{blue}{8} \\ \color{green}{10} & \color{purple}{12} \end{bmatrix}\]

You Try: Matrix Addition

Problem: Add these matrices \(\mathbf{A} = \begin{bmatrix} 3 & -1 & 2 \\ 0 & 4 & -3 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 2 & -1 \\ 5 & -2 & 1 \end{bmatrix}\)

You Try: \(\mathbf{A} + \mathbf{B} = ?\)

04:00

Let’s verify in R


A <- matrix(c(3, -1, 2,
              0, 4, -3), 
            nrow = 2, byrow = TRUE)
B <- matrix(c(1, 2, -1,
              5, -2, 1),
            nrow = 2, byrow = TRUE)
A + B
     [,1] [,2] [,3]
[1,]    4    1    1
[2,]    5    2   -2

Matrix Addition: Dimension Requirements

This works

\[\colorbox{#eec8e1}{$\mathbf{A}_{2 \times 3} + \mathbf{B}_{2 \times 3} = \mathbf{C}_{2 \times 3}$}\]

This doesn’t work

\[\colorbox{#eec8e1}{$\mathbf{A}_{2 \times 3} + \mathbf{B}_{3 \times 2} = \text{?}$}\]

The matrices must have exactly the same dimensions to be added!

Matrix Multiplication: The Basics

  • Matrix multiplication is NOT element-wise multiplication
  • For \(\mathbf{A} \times \mathbf{B}\) to be defined the number of columns in A must equal number of rows in B (inner dimensions must match!)
  • If \(\mathbf{A}\) is \(m \times n\) and \(\mathbf{B}\) is \(n \times p\), then: \[\mathbf{A}_{m \times n} \times \mathbf{B}_{n \times p} = \mathbf{C}_{m \times p}\]

You Try: Matrix Dimensions

Problem: Can these matrices be multiplied? If yes, what’s the resulting dimension?

\(\mathbf{A}_{3 \times 2} \times \mathbf{B}_{2 \times 4} = ?\)

\(\mathbf{C}_{2 \times 5} \times \mathbf{D}_{3 \times 2} = ?\)

Solutions:

  1. \(\mathbf{A}_{3 \times 2} \times \mathbf{B}_{2 \times 4} = \mathbf{Result}_{3 \times 4}\) ✓ (inner dimensions match: 2 = 2)

  2. \(\mathbf{C}_{2 \times 5} \times \mathbf{D}_{3 \times 2}\) = undefined ✗ (inner dimensions don’t match: 5 ≠ 3)

02:00

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}&\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{}&\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[\begin{bmatrix} \colorbox{#eec8e1}{1} & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{(1)(5)}&\\&\end{bmatrix}\]

Matrix Multiplication: Step by Step

\[\begin{bmatrix} \colorbox{#eec8e1}{1} & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{(1)(5) + (2)(7)}&\\&\end{bmatrix}\]

Matrix Multiplication: Step by Step

\[\begin{bmatrix} \colorbox{#eec8e1}{1} & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{19}&\\&\end{bmatrix}\]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{ }\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} \colorbox{#eec8e1}1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{(1)(6)}\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} \colorbox{#eec8e1}1 & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{(1)(6)+(2)(8)}\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} \colorbox{#eec8e1}1 & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{22}\\&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{ }&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{(3)(5) }&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{(3)(5) +(4)(7)}&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{43}&\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{ }\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{(3)(6) }\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{(3)(6)+(4)(8) }\end{bmatrix} \]

Matrix Multiplication: Step by Step

\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{52}\end{bmatrix} \]

You Try: Matrix Multiplication

Problem: Multiply these matrices \(\mathbf{A} = \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 2 & 1 \end{bmatrix}\)

You Try: \(\mathbf{A} \times \mathbf{B} = ?\)

Solution:

\(\mathbf{A} \times \mathbf{B} = \begin{bmatrix} 8 & 3 \\ 9 & 4 \end{bmatrix}\)

05:00

Let’s verify in R


A <- matrix(c(2, 3,
              1, 4), 
            nrow = 2, byrow = TRUE)
B <- matrix(c(1, 0,
              2, 1),
            nrow = 2, byrow = TRUE)
A %*% B
     [,1] [,2]
[1,]    8    3
[2,]    9    4

Matrix Multiplication: Dimension Examples

Valid multiplications:

  • \(\mathbf{A}_{2 \times 3} \times \mathbf{B}_{3 \times 4} = \mathbf{C}_{2 \times 4}\)

  • \(\mathbf{A}_{5 \times 2} \times \mathbf{B}_{2 \times 1} = \mathbf{C}_{5 \times 1}\)

Invalid multiplications:

  • \(\mathbf{A}_{2 \times 3} \times \mathbf{B}_{4 \times 2}\) ✗ (3 ≠ 4)

  • \(\mathbf{A}_{3 \times 5} \times \mathbf{B}_{2 \times 3}\) ✗ (5 ≠ 2)

Remember: Inner dimensions must match!

Matrix Transpose

The transpose of matrix \(\mathbf{A}\) is denoted \(\mathbf{A}^T\) or \(\mathbf{A}'\)

Flip rows and columns

  • Row 1 becomes Column 1
  • Row 2 becomes Column 2, etc.

Matrix Transpose

\[\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}_{2 \times 3}\]

\[\mathbf{A}^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}_{3 \times 2}\]

You Try: Matrix Transpose

Find the transpose of this matrix \(\mathbf{A} = \begin{bmatrix} 1 & 4 & 7 \\ 2 & 5 & 8 \end{bmatrix}\) \(A^T= ?\)

Solution: \(\mathbf{A}^T = \begin{bmatrix} 1 & 2 \\ 4 & 5 \\ 7 & 8 \end{bmatrix}\)

03:00

Let’s verify in R


A <- matrix(c(1, 4, 7,
              2, 5, 8), 
            nrow = 2, byrow = TRUE)

t(A)
     [,1] [,2]
[1,]    1    2
[2,]    4    5
[3,]    7    8

Transpose Properties

If \(\mathbf{A}\) is \(m \times n\), then \(\mathbf{A}^T\) is \(n \times m\)

Transpose Properties

  1. \((\mathbf{A}^T)^T = \mathbf{A}\)

  2. \((\mathbf{A} + \mathbf{B})^T = \mathbf{A}^T + \mathbf{B}^T\)

  3. \((\mathbf{AB})^T = \mathbf{B}^T\mathbf{A}^T\) (order reverses!)

  4. Symmetric Matrix: \(\mathbf{A} = \mathbf{A}^T\)

Symmetric Matrix

\(\mathbf{A} = \mathbf{A}^T\)

\[\mathbf{S} = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 5 \\ 3 & 5 & 6 \end{bmatrix}\]

The Identity Matrix

The identity matrix \(\mathbf{I}\) is the matrix equivalent of the number 1

  • Square matrix with 1’s on the diagonal and 0’s elsewhere

  • Acts as the multiplicative identity: \(\mathbf{AI} = \mathbf{IA} = \mathbf{A}\)

The Identity Matrix

\[\mathbf{I}_2 = \begin{bmatrix} \color{red}{1} & 0 \\ 0 & \color{red}{1} \end{bmatrix}, \quad \mathbf{I}_3 = \begin{bmatrix} \color{red}{1} & 0 & 0 \\ 0 & \color{red}{1} & 0 \\ 0 & 0 & \color{red}{1} \end{bmatrix}\]

The Identity Matrix

Example: \[\begin{bmatrix} 2 & 3 \\ 4 & 5 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 4 & 5 \end{bmatrix}\]

You Try: Identity Matrix

Problem: Verify that this multiplication gives the identity property \(\begin{bmatrix} 3 & -1 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = ?\)

Solution: \(= \begin{bmatrix} 3 & -1 \\ 2 & 4 \end{bmatrix}\) ✓ (Original matrix unchanged!)

03:00

Let’s verify in R


A <- matrix(c(3, -1,
              2, 4), 
            nrow = 2, byrow = TRUE)
I <- diag(2)
A %*% I
     [,1] [,2]
[1,]    3   -1
[2,]    2    4

Matrix Inverse

The inverse of matrix \(\mathbf{A}\) is denoted \(\mathbf{A}^{-1}\)

Definition: \(\mathbf{AA}^{-1} = \mathbf{A}^{-1}\mathbf{A} = \mathbf{I}\)

Matrix Inverse

Requirements for inverse to exist:
- Matrix must be square (n × n)
- Matrix must be non-singular (determinant ≠ 0)

Matrix Inverse

2 × 2 Formula: For \(\mathbf{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\)

\[\mathbf{A}^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}\]

Matrix Inverse: Example

\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]

Step 1: Calculate determinant \[\text{det}(\mathbf{A}) = (2)(2) - (1)(6) = 4 - 6 = -2\]

Matrix Inverse: Example

\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]

Step 2: Apply formula \[\mathbf{A}^{-1} = \frac{1}{2} \begin{bmatrix} 2 & -1 \\ -6 & 2 \end{bmatrix} = \begin{bmatrix} 1 & -0.5 \\ -3 & 1 \end{bmatrix}\]

Matrix Inverse: Example

\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]

Verify: \[\mathbf{AA}^{-1} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix} \begin{bmatrix} 1 & -0.5 \\ -3 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \mathbf{I}\]

You Try: Matrix Inverse

Problem: Find the inverse of this matrix \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 4 & 3 \end{bmatrix}\)

Solution: 1. \(\text{det}(\mathbf{A}) = (1)(3) - (2)(4) = 3 - 8 = -5\)
2. \(\mathbf{A}^{-1} = \frac{1}{-5} \begin{bmatrix} 3 & -2 \\ -4 & 1 \end{bmatrix} = \begin{bmatrix} -0.6 & 0.4 \\ 0.8 & -0.2 \end{bmatrix}\)

Check: \(\mathbf{AA}^{-1} = \begin{bmatrix} 1 & 2 \\ 4 & 3 \end{bmatrix} \begin{bmatrix} -0.6 & 0.4 \\ 0.8 & -0.2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\)

06:00

Let’s verify in R


A <- matrix(c(1, 2, 
              4, 3), 
            nrow = 2, byrow = TRUE)
solve(A)
     [,1] [,2]
[1,] -0.6  0.4
[2,]  0.8 -0.2
round(A %*% solve(A), 10)
     [,1] [,2]
[1,]    1    0
[2,]    0    1

What about matrices of higher dimension?

  • Life is short! Let’s just use R.
  • Be sure to check the determinant! (it cannot be 0)
A <- matrix(c(1, 2,
              2, 4),
            nrow = 2, byrow = TRUE)
det(A)
[1] 0

Special Matrix Types

Diagonal Matrix: Non-zero elements only on the main diagonal

\[\mathbf{D} = \begin{bmatrix} d_1 & 0 & 0 \\ 0 & d_2 & 0 \\ 0 & 0 & d_3 \end{bmatrix}\]

Special Matrix Types

Orthogonal Matrix: \(\mathbf{Q}^T\mathbf{Q} = \mathbf{I}\) (columns are orthonormal)

Special Matrix Types

Positive Definite: \(\mathbf{x}^T\mathbf{A}\mathbf{x} > 0\) for all \(\mathbf{x} \neq \mathbf{0}\)

Special Matrix Types

Symmetric Matrix: \(\mathbf{A} = \mathbf{A}^T\)

You Try: Special Matrices

Problem: Identify the type of each matrix \(\mathbf{A} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 4 \\ 4 & 2 \end{bmatrix}, \quad \mathbf{C} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\)

Solutions: - Matrix A: Diagonal matrix (non-zero elements only on main diagonal) - Matrix B: Symmetric matrix (\(\mathbf{B} = \mathbf{B}^T\) since \(b_{12} = b_{21} = 4\)) - Matrix C: General matrix (no special properties)

02:00

Vector and Matrix Derivatives

Vector Derivatives: Introduction

Why do we need derivatives of vectors? - Essential for optimization in statistics and machine learning - Key to deriving least squares solutions - Foundation for understanding the hat matrix

Vector Derivatives: Notation

Let \(\mathbf{a} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}\) and \(\mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}\) be column vectors

Derivative Rule: Linear Form

Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{a}'\mathbf{x}) = \mathbf{a}\)

In words: The derivative of a linear form \(\mathbf{a}'\mathbf{x}\) with respect to \(\mathbf{x}\) is simply \(\mathbf{a}\)

Derivative Rule: Example

Let \(\mathbf{a} = \begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}\)

Then \(\mathbf{a}'\mathbf{x} = 2x_1 + 3x_2 + x_3\)

Derivative: \(\frac{\partial}{\partial \mathbf{x}}(2x_1 + 3x_2 + x_3) = \begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix} = \mathbf{a}\)

You Try: Vector Derivatives

Problem: Find \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{c}'\mathbf{x})\) where \(\mathbf{c} = \begin{bmatrix} 5 \\ -2 \\ 4 \end{bmatrix}\)

Solution: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{c}'\mathbf{x}) = \mathbf{c} = \begin{bmatrix} 5 \\ -2 \\ 4 \end{bmatrix}\)

03:00

Matrix Derivatives: Quadratic Forms

Quadratic form: \(\mathbf{x}'\mathbf{A}\mathbf{x}\) where \(\mathbf{A}\) is an \(n \times n\) matrix

Example: \(\mathbf{x}'\mathbf{A}\mathbf{x} = \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)

Matrix Derivatives: Quadratic Forms

Expanding the 2×2 case: \[\mathbf{x}'\mathbf{A}\mathbf{x} = a_{11}x_1^2 + (a_{12} + a_{21})x_1x_2 + a_{22}x_2^2\]

Derivative Rule 3: Quadratic Forms

Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}')\mathbf{x}\)

Special case: If \(\mathbf{A}\) is symmetric (\(\mathbf{A} = \mathbf{A}'\)), then: \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = 2\mathbf{A}\mathbf{x}\]

Derivative Rule 3: Example

Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)

Step 1: Find \(\mathbf{A} + \mathbf{A}'\) \[\mathbf{A}' = \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \mathbf{A} + \mathbf{A}' = \begin{bmatrix} 4 & 4 \\ 4 & 8 \end{bmatrix}\]

Derivative Rule 3: Example (cont.)

Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)

Step 2: Apply the rule \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}')\mathbf{x} = \begin{bmatrix} 4 & 4 \\ 4 & 8 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4x_1 + 4x_2 \\ 4x_1 + 8x_2 \end{bmatrix}\]

Symmetric Matrix Case

Example: Let \(\mathbf{S} = \begin{bmatrix} 3 & 2 \\ 2 & 5 \end{bmatrix}\) (symmetric)

Derivative: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{S}\mathbf{x}) = 2\mathbf{S}\mathbf{x} = 2\begin{bmatrix} 3 & 2 \\ 2 & 5 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 6x_1 + 4x_2 \\ 4x_1 + 10x_2 \end{bmatrix}\)

General Form: \(\mathbf{b}'\mathbf{A}\mathbf{x}\)

Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{b}'\mathbf{A}\mathbf{x}) = \mathbf{A}'\mathbf{b}\)

Note: \(\mathbf{b}'\mathbf{A}\mathbf{x}\) is a scalar, and \(\mathbf{b}\) is treated as a constant vector

You Try: Matrix Derivatives

Problem: Find \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{B}\mathbf{x})\) where \(\mathbf{B} = \begin{bmatrix} 1 & 0 \\ 2 & 3 \end{bmatrix}\)

Solution: \[\mathbf{B}' = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}, \quad \mathbf{B} + \mathbf{B}' = \begin{bmatrix} 2 & 2 \\ 2 & 6 \end{bmatrix}\] \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{B}\mathbf{x}) = \begin{bmatrix} 2 & 2 \\ 2 & 6 \end{bmatrix} \mathbf{x} = \begin{bmatrix} 2x_1 + 2x_2 \\ 2x_1 + 6x_2 \end{bmatrix}\]

04:00

Summary: Key Rules

  1. Linear forms: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{a}'\mathbf{x}) = \mathbf{a}\)
  1. Quadratic forms: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}')\mathbf{x}\)
  1. Symmetric case: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = 2\mathbf{A}\mathbf{x}\) when \(\mathbf{A} = \mathbf{A}'\)

Connection to Hat Matrix

These rules are essential for deriving:

  • Normal equations: \(\mathbf{X}'\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}'\mathbf{y}\)

  • Hat matrix: \(\mathbf{H} = \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\)

  • Least squares solution: \(\hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{y}\)

We’ll use these derivatives to minimize the sum of squared errors!